Fast Spectral Low Rank Matrix Approximation

In this paper, we study subspace embedding problem and obtain the following results: 1. We extend the results of approximate matrix multiplication from the Frobenius norm to the spectral norm. Assume matrices A and B both have at most r stable rank and r̃ rank, respectively. Let S be a subspace embedding matrix with l rows which depends on stable rank, then with high probability, we have ‖ASSB−AB‖2 < ε‖A‖2‖B‖2. 2. We develop a class of fast approximate generalized linear regression algorithms with respect to the spectral norm. We design a new least square regression algorithm in which subspace embedding matrix S has ( √ ε/r, δ)-JL moment property. Here r is the stable rank A, which is never greater than rank of A. Let x = argmin x ‖SAx− Sb‖2, we have ‖Ax − b‖2 ≤ (1 + ε)min x ‖Ax− b‖2. 3. We give a concise proof and tighter error upper bound for the randomized SVD of Halko et al. (2011). Besides gaussian random projection and Subsample Randomized Hadamard Transform in Halko et al. (2011), we find that a large class of matrices which have oblivious l2-subspace embedding property can be used in randomized SVD. We give a fast randomized SVD algorithm using sparse embedding matrix. We give a framework that composing different subspace embedding matrices still has the same relative error bound. 4. We design a fast low rank approximation algorithm with relative error based on spectral norm and the stable rank. For A ∈ R, given k, and ε, we get a decomposition of A into L, D, W, such that ‖A− LDW ‖2 ≤ (1 + ε)‖A−Ak‖2, and our algorithm runs in Õ(nnz(A)ε+(n+ d)r 1 /ε+ r1r 2 2 /ε). A2k and Ad/k both have stable rank at most r1. SA and A−A(SA)†SA both have stable rank at most r2. And S is a sparse subspace embedding matrix with Õ(r1/ε) rows.